PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

We design a compact but effective CNN model for optical flow by exploiting the well-known design principles: pyramid, warping, and cost volume. Cast in a learnable feature pyramid, our network uses the current optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct the cost volume, which is processed by a CNN network to decode the optical flow. As the cost volume is a more discriminative representation of the search space for the optical flow than raw images, a compact CNN decoder network is sufficient. Our model performs on par with the recent FlowNet2 method on the MPI Sintel and KITTI 2015 benchmarks, while being 17 times smaller in size and 2 times faster in inference. Our model protocol and learned parameters will be publicly available.